Detection of driving actions that mitigate risk

ABSTRACT

Systems and methods are provided for detecting a driving action that mitigates risk.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of PCT/US2018/55631, filed on the 12Oct. 2018, and titled “DETECTION OF DRIVING ACTIONS THAT MITIGATE RISK,”which claims the benefit of U.S. Provisional Patent Application No.62/571,617 filed on 12 Oct. 2017, and titled, “SYSTEM AND METHODS OFGENERATING A TRAFFIC INCIDENT REPORT”, and U.S. Provisional PatentApplication No. 62/573,120 filed on 16 Oct. 2017, and titled, “DETECTIONOF DRIVING ACTIONS THAT MITIGATE RISK”, the disclosures of which areexpressly incorporated by reference in their entireties.

BACKGROUND Field

Certain aspects of the present disclosure generally relate tointelligent driving monitoring systems (IDMS), driver monitoringsystems, advanced driver assistance systems (ADAS), and autonomousdriving systems, and more particularly to systems and methods fordetecting driving actions that mitigate risk and systems and methods forthe detection of driving actions that mitigate risk.

Background

Vehicles, such as automobiles, trucks, tractors, motorcycles, bicycles,airplanes, drones, ships, boats, submarines, and others, are typicallyoperated and controlled by human drivers. Through training and withexperience, a human driver may learn how to drive a vehicle safely andefficiently in a range of conditions or contexts. For example, as anautomobile driver gains experience, he may become adept at driving inchallenging conditions such as rain, snow, or darkness.

Drivers may sometimes drive unsafely or inefficiently. Unsafe drivingbehavior may endanger the driver and other drivers and may risk damagingthe vehicle. Unsafe driving behaviors may also lead to fines. Forexample, highway patrol officers may issue a citation for speeding.Unsafe driving behavior may also lead to accidents, which may causephysical harm, and which may, in turn, lead to an increase in insurancerates for operating a vehicle. Inefficient driving, which may includehard accelerations, may increase the costs associated with operating avehicle.

Driving behavior may be monitored. Driver monitoring may be done inreal-time as the driver operates a vehicle, or may be done at a latertime based on recorded data. Driver monitoring at a later time may beuseful, for example, when investigating the cause of an accident. Drivermonitoring in real-time may be useful to guard against unsafe driving,for example, by ensuring that a car cannot exceed a certainpre-determined speed. The types of monitoring available today, however,may be based on sensors that do not provide context to a traffic event.For example, an accelerometer may be used to detect a suddendeceleration associated with a hard-stopping event, but theaccelerometer may not be aware of the cause of the hard-stopping event.

Prior approaches to driver monitoring may be based on the occurrence ofnegative driving events, such as hard-braking or speeding, and may notconsider positive measures, such as determinations that a driver'sbehavior contributed to the avoidance of an unsafe traffic situation.Accordingly, certain aspects of the present disclosure are directed todetecting positive driving actions, such as driving actions thatmitigate risk.

SUMMARY

Certain aspects of the present disclosure generally relate to providing,implementing, and using a method of detecting driving actions thatmitigate risk. The methods may involve a camera sensor and/or inertialsensors to detect traffic events, as well analytical methods that maydetermine an action by a monitored driver that is responsive to thedetected traffic event.

Certain aspects of the present disclosure provide a system. The systemgenerally includes a memory and a processor coupled to the memory. Theprocessor is configured to: determine an occurrence of an atypicaltraffic event at or near a monitored vehicle; and determine an actionresponsive to the atypical traffic event by a driver or a control systemof the monitored vehicle based on data collected at the monitoredvehicle.

Certain aspects of the present disclosure provide a non-transitorycomputer readable medium having instructions stored thereon. Uponexecution, the instructions cause the computing device to performoperations comprising: determining an occurrence of an atypical trafficevent at or near a monitored vehicle; and determining an actionresponsive to the atypical traffic event by a driver or a control systemof the monitored vehicle based on data collected at the monitoredvehicle.

Certain aspects of the present disclosure provide a method. The methodgenerally includes determining, by a processor of a computing device, anoccurrence of an atypical traffic event at or near a monitored vehicle;and determining, by the processor, an action responsive to the atypicaltraffic event by a driver or a control system of the monitored vehiclebased on data collected at the monitored vehicle.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example of a system for detecting a driving actionthat mitigates risk in accordance with certain aspects of the presentdisclosure.

FIG. 2 illustrates an example of a driver monitoring system inaccordance with certain aspects of the present disclosure.

FIG. 3A illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 3B illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 3C illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 3D illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 4A illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 4B illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 4C illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 4D illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 5A illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 5B illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 5C illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 5D illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 6A illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 6B illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 6C illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 6D illustrates an example of a system for detecting a drivingaction that mitigates risk in accordance with certain aspects of thepresent disclosure.

FIG. 7 illustrates an example of a system for automatically generating areport of a traffic event in accordance with certain aspects of thepresent disclosure.

DETAILED DESCRIPTION

The detailed description set forth below, in connection with theappended drawings, is intended as a description of variousconfigurations and is not intended to represent the only configurationsin which the concepts described herein may be practiced. The detaileddescription includes specific details for the purpose of providing athorough understanding of the various concepts. However, it will beapparent to those skilled in the art that these concepts may bepracticed without these specific details. In some instances, well-knownstructures and components are shown in block diagram form in order toavoid obscuring such concepts.

Based on the teachings, one skilled in the art should appreciate thatthe scope of the disclosure is intended to cover any aspect of thedisclosure, whether implemented independently of or combined with anyother aspect of the disclosure. For example, an apparatus may beimplemented or a method may be practiced using any number of the aspectsset forth. In addition, the scope of the disclosure is intended to coversuch an apparatus or method practiced using other structure,functionality, or structure and functionality in addition to or otherthan the various aspects of the disclosure set forth. It should beunderstood that any aspect of the disclosure disclosed may be embodiedby one or more elements of a claim.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any aspect described herein as “exemplary”is not necessarily to be construed as preferred or advantageous overother aspects.

Although particular aspects are described herein, many variations andpermutations of these aspects fall within the scope of the disclosure.Although some benefits and advantages of the preferred aspects arementioned, the scope of the disclosure is not intended to be limited toparticular benefits, uses or objectives. Rather, aspects of thedisclosure are intended to be broadly applicable to differenttechnologies, system configurations, networks and protocols, some ofwhich are illustrated by way of example in the figures and in thefollowing description of the preferred aspects. The detailed descriptionand drawings are merely illustrative of the disclosure rather thanlimiting, the scope of the disclosure being defined by the appendedclaims and equivalents thereof.

Monitoring and Characterization of Driver Behavior

Aspects of the present disclosure are directed to methods of monitoringand characterizing driver behavior, which may include methods ofdetecting a driving action that mitigates risk. An accuratecharacterization of driver behavior has multiple applications. Insurancecompanies may desire aggregated driver behavior data to influencepremiums. Insurance companies, fleet managers, and the like, may seek toreward safe driving behavior and dis-incentivize unsafe drivingbehaviors, for example, as a means to reducing the number of loss eventsacross a population of drivers. In addition, it may be desirable toreward driving behaviors that avoid or ameliorate the occurrence ofunsafe driving scenarios. Furthermore, fleet owners might desire asystem capable of classifying driver behaviors as a part of a program toincentivize their drivers to drive safely and efficiently. Taxiaggregators may desire a driver monitoring system as part of a programto incentivize taxi driver behavior, and/or taxi or ride-sharingaggregator customers may desire access to past characterizations ofdriver behavior. With knowledge of driver behavior, customers may filterand/or select drivers based on driver behavior criteria. For example, toensure safety, drivers of children or other vulnerable populations maybe screened based on driving behavior exhibited in the past. Parents maydesire to monitor the driving patterns of their kids and may furtherutilize methods of monitoring and characterizing driver behavior toincentivize safe driving behavior.

In addition to human drivers, machine controllers are increasingly beingused to drive vehicles. Self-driving cars, for example, may include amachine controller (which may be referred to as a computerized drivingcontroller) that interprets sensory inputs and issues control signals tothe car so that the car may be driven without a human driver or withminimal human intervention. As with human drivers, machine controllersmay also exhibit unsafe or inefficient driving behaviors. Informationrelating to the driving behavior of a self-driving car would be ofinterest to engineers attempting to perfect the self-driving car'scontroller, to law-makers considering policies relating to self-drivingcars, and to other interested parties.

Visual information may improve existing ways or enable new ways ofmonitoring and characterizing driver behavior. For example, according toaspects of the present disclosure, the visual environment around adriver may inform a characterization of driver behavior. Typically,running a red light may be considered an unsafe driving behavior. Insome contexts, however, such as when a traffic guard is standing at anintersection and using hand gestures to instruct a driver to movethrough a red light, driving through a red light would be considered asafe and/or compliant driving behavior. Additionally, in some contexts,an unsafe driving behavior, such as tailgating, may not be the fault ofthe driver. For example, another driver may have pulled into thedriver's lane at an unsafe distance ahead of the driver. Visualinformation may also improve the quality of a characterization that maybe based on other forms of sensor data, such as determining a safedriving speed, as described below.

The costs of accurately characterizing driver behavior on an enableddevice that is coupled to the driver's vehicle, using computer visionmethods in accordance with certain aspects of the present disclosure,may be less than the costs of alternative methods that use humaninspection of visual data. Camera based methods may have lower hardwarecosts compared with methods that involve RADAR or LiDAR. Still, methodsthat use RADAR or LiDAR are also contemplated for determination of causeof traffic events, either alone or in combination with a vision sensor,in accordance with certain aspects of the present disclosure.

FIG. 1 illustrates an embodiment of the aforementioned system fordetecting a driving action that mitigates risk. The device 100 mayinclude input sensors (which may include a forward-facing camera 102, adriver facing camera 104, connections to other cameras that are notphysically mounted to the device, inertial sensors 106, car OBD-II portsensor data (which may be obtained through a Bluetooth connection 108),and the like) and compute capability 110. The compute capability may bea CPU or an integrated System-on-a-chip (SOC), which may include a CPUand other specialized compute cores, such as a graphics processor (GPU),gesture recognition processor, and the like. In some embodiments, asystem for detecting a driving action that mitigates risk may includewireless communication to cloud services, such as with Long TermEvolution (LTE) 116 or Bluetooth communication 108 to other devicesnearby. For example, the cloud may provide real-time analyticsassistance. In an embodiment involving cloud services, the cloud mayfacilitate aggregation and processing of data for offline analytics. Thedevice may also include a global positioning system (GPS) either as aseparate module 112, or integrated within a System-on-a-chip 110. Thedevice may further include memory storage 114.

A system for detecting a driving action that mitigates risk, inaccordance with certain aspects of the present disclosure, may assessthe driver's behavior in real-time. For example, an in-car monitoringsystem, such as the device 100 illustrated in FIG. 1 that may be mountedto a car, may perform analysis in support of a driver behaviorassessment in real-time, may determine cause of traffic events as theyoccur, and may recognize that a driver's action was responsive to anexisting or pending traffic situation that served to reduce risk. Inthis example, the system, in comparison with a system that does notinclude real-time processing, may avoid storing large amounts of sensordata since it may instead store a processed and reduced set of the data.Similarly, or in addition, the system may incur fewer costs associatedwith wirelessly transmitting data to a remote server.

A system for detecting a driving action that mitigates risk, inaccordance with certain aspects of the present disclosure, may assessthe driver's behavior in several contexts and perhaps using severalmetrics. FIG. 2 illustrates a system of driver monitoring, which mayinclude a system for detecting a driving action that mitigates risk, inaccordance with aspects of the present disclosure. The system mayinclude sensors 210, profiles 230, sensory recognition and monitoringmodules 240, assessment modules 260, and may produce an overall grade280. Contemplated driver assessment modules include speed assessment262, safe following distance 264, obeying traffic signs and lights 266,safe lane changes and lane position 268, hard accelerations includingturns 270, responding to traffic officers, responding to road conditions272, and responding to emergency vehicles. Each of these exemplaryfeatures is described in PCT application PCT/US17/13062, entitled“DRIVER BEHAVIOR MONITORING”, filed 11 Jan. 2017, which is incorporatedherein by reference in its entirety. The present disclosure is not solimiting, however. Many other features of driving behavior may bemonitored, assessed, and characterized in accordance with the presentdisclosure.

Safe Following Distance

Aspects of the present disclosure are directed to visually measuring afollowing distance 264, which is a distance to a vehicle directly infront of a driver's car. Several methods of visually measuring thefollowing distance are contemplated. For example, a mono-camera 102 maybe used to identify the type of vehicle being followed, such as a sedan,van, or semi-truck. In this example, the following distance may be basedon feature sizes, such as width, or the relative feature sizes ofmultiple features associated with each type of vehicle. In anotherexample, a machine learning model, such as a deep neural network, may beused to determine the distance based on the input pixels correspondingto the vehicle ahead. While the preceding examples utilize amono-camera, the present disclosure is not so limiting. In anotherexample, multiple cameras and/or other sensors, such as RADAR,Ultrasound (SONAR), or LiDAR, may be used to determine the distance tothe vehicle ahead. In addition, multiple methods may be combined toestimate the distance.

In an embodiment of the present disclosure, a driver monitoring systemmay determine the speed of the driver's vehicle 246 and the speed of thevehicle ahead 248. The system may then assess the driver's safefollowing behavior 264, and determine a safe following grade as afunction of the distance to the car and the speeds of the vehicles. Inaddition, the system may further determine the speed of other traffic248 and may incorporate the speed of other traffic in the assessment ofthe driver's safe following behavior 268.

In another embodiment of the aforementioned driver monitoring system,the determined following distance may be converted from a unit ofdistance, such as from feet or meters, to a unit of time, such asseconds. In this example, the assessment of safe following behavior 264may be based on this inferred measure of following time. The drivergrade for safe following behavior may be computed as a function of thefollowing distance in time, and may also be based on the estimatedstopping time based on the current speed of the car 246. For example,driving with less than 0.7 seconds following time while travelling 30MPH, or driving with less than 1 second following time while travelling65 MPH may result in a reduction in the driver's safe following grade.Other threshold values may be used depending on the safety and/or fuelefficiency goals of a driver. Threshold values may be set, for example,by a safety manager of a vehicle fleet. For example, a system may beconfigured such that driving with less than 2 seconds following timewhile travelling 30 MPH, or driving with less than 5 seconds followingtime while travelling 65 MPH may result in a reduction in the driver'ssafe following grade. Safe following thresholds may also be based, atleast in part, on weather conditions, posted speed limits, or customaryspeeds and/or following distances for a given road and/or for a givenregion.

In the present example, a method of determining the following distancein accordance with the present disclosure may involve a computer visionmodel. For example, the determination of following distance may involverecognizing the type of vehicle ahead along with the make and model ofthe vehicle, determining dimensions of the vehicle based on the make andmodel, computing the observed dimensions of the vehicle (e.g. in pixelcoordinates), and estimating the distance based on the relationshipbetween the observed dimensions in the visual data and known vehicledimensions in real-world coordintates. Similarly, a computer visionmodel may be based on the detection of the vehicle without recognizingthe make and model of the vehicle, and estimating the distance based ona relationship between the observed and known average or median vehicledimensions of that type. Alternatively, a neural network, such as a deepneural network, may be trained on a set of distance estimates associatedwith stored sensor inputs. A neural network trained on such inputs maythen output an estimate of following distance to a detected car based ona new set of inputs.

Safe Lane Changes and Lane Position

Aspects of the present disclosure may be directed to assessing thequality of lane changes and lane position 268. For example, the drivermonitoring system may use either visual 212, RADAR, LiDAR, or othersystems 210 to determine the relative positions of vehicles around thecar. The driver monitoring system may then assess the driver's aptitudein maintaining a safe location, such as not driving next to cars inadjacent lanes, but rather maintaining an offset in position. Duringlane changes, the driver monitoring system may assess a characteristicof the driver's driving behavior (such as ability, safety, and the like)based on the relative distances and speeds of the driver's car 246 andnearby cars 248 when changing lanes. In addition, this assessment may bebased on whether and when the driver signaled lane changes, which may beaccessed via the OBD-II 226, and/or with the aid of a microphone thatcan detect and recognize the sound of a turn signal indicator.

The driver monitoring system may also determine the rate of closure ofcars in adjacent lanes and use that rate of closure to modify the driverassessment. For example, if a driver changed into a lane with a fastapproaching car, the distance threshold for a safe assessment of thelane change may be greater than it would have been if the approachingcar were going about the same speed as the driver's car.

In another example, a monitored driver may notice that a second car isabout to change lanes into his lane, and based on his experience, mayknow that when the second driver's lane change is completed, thefollowing distance between the monitored driver's car and the second carwill be dangerously short. This may create a scenario in which theability of the monitored driver to react with enough time to avoid acollision may be reduced, which may be critical if traffic flow were tostop suddenly or the vehicle in front of the monitored driver were tostop or suddenly slow.

In accordance with certain aspects of the present disclosure, a systemmay determine a dangerously short following distance is likely to occurin the near future. The system may then detect whether the driver makesany action that is responsive to this predicted event. In this example,the monitored driver may slow down or change lanes and thereby increasethe distance between himself and the second car.

In some embodiments of the present disclosure, a system or method mayassess the risk of the driving environment after the driver performed aresponsive action. The assessed risk may then be compared with a riskassessment of the driving environment before the driver performed theaction. Alternatively, or in addition, the assessed risk may be comparedwith a risk assessment of the driving environment that the systempredicts would have occurred had the driver not performed the action.

In the present example, the monitored driver may have deacceleratedbefore the second car entered his lane. In this case, the risk levelbefore the driver's action may actually be lower than the risk levelafter the driver's action, since the monitored driver may be tailgatingthe second driver shortly after his responsive action, but not before.For this reason, it may be more informative to compare a drivingenvironment risk after the driver's action with a level of risk thatwould have occurred had the driver not performed the action. In thisexample, if the driver had not deaccelerated, the following distancewould have been even less, and the driving environment correspondinglyriskier. Additional examples of tailgating-type events, includingavoided tailgating events, and responsive actions are described below.

Based on a determination that a driver's action mitigated risk in thedriving environment, a system or method in accordance with the presentdisclosure may generate an alert notification. In one example, data maybe communicated from a device coupled to the monitored driver's vehicleto a second device. The communicated data may include an inference thatthe driver performed an action that mitigated risk. In some embodiments,the cloud server may then automatically request additional data, such asvideo data, from the device. The additional data may be used torecognize the driver's proactive driving, may be used as a trainingvideo for other driver's, may be used to train an autonomous drivingsystem, and the like. The communicated data may be referred to as a“StarAlert”, a DriverStar, and the like. In some embodiments, a“StarAlert” may comprise data requested by a cloud server.

A ‘StarAlert’ or DriverStar may include an identification of the drivercreating space between his vehicle and the vehicle directly in front ofhis vehicle. In one example, the system may determine that a secondvehicle is about to enter the driver's lane from a merging lane of ahighway. The determination may be based on detectable attributes of lanelines and/or road boundaries, such as the presence of a dotted lane linepattern, the termination of such a lane pattern, or a curve in a roadboundary indicative of the end of a merging zone. In addition, oralternatively, the system may determine that the second vehicle is aboutto enter the driver's lane because the system is operating on datawithin a temporal buffer of several seconds (e.g. 10 seconds), andbecause a separate processing thread has determined that the secondvehicle did in fact enter the driver's lane a few seconds ahead of thetime of interest. The system may then determine if the monitored driverexhibited any proactive behaviors in response to the merging driver. Forexample, the driver's vehicle may have slowed by 5 mph within a periodleading up to or just after the second vehicle entered his lane.Furthermore, the system may determine that the distance between thedriver's vehicle and the second vehicle has increased around the sametime, such that a tailgating event was substantially avoided. Such aproactive driving maneuver may be automatically detected. In someembodiments, the driving maneuver just described may be referred to as a“Driver Star—Create Separation”, reflecting the outcome of the driver'sreduction in speed—which was to create additional separation between themonitored driver and the merging vehicle

The driver may have performed one or more of a number of driving actionsto generate a “StarAlert”. The responsive driving action could be acombination of one or more of the following; reduced speed, reducedrelative speed to corresponding traffic, deactivation of throttle, oractivation of brake pedal. Each of these actions, alone or incombination, may correspond to the monitored driver taking a proactivedriving action to mitigate the risky driving condition unfolding beforehim.

Advanced Path Prediction

Certain aspects of the present disclosure may include Advanced PathPrediction (APP). Systems and methods of advanced path prediction aredescribed in PCT application PCT/US17/23781—“Advanced Path Prediction”,filed 23 Mar. 2017, which is incorporated herein by reference in itsentirety. According to certain aspects, the path of travel may be mappedto a camera view, and/or may incorporate information from later pointsin time.

APP may be applied to systems that determine driving behaviors inresponse to objects in the environment. Tailgating, for example, is adriving behavior in which a Driver drives at an unsafe distance behindthe car ahead of it in the same lane. Since a determination oftailgating behavior depends on a determination of the lane occupied byeach of two cars, a method for determining a lane of travel may bedesirable for a system for detecting a driving action that mitigatesrisk. A driving monitoring system with robust lane detection, forexample, may be able to properly ignore situations that may appear to betailgating but in which the car ahead is actually in a different lane,which might be the case on a curved road having multiple lanes oftraffic.

Given the determined object locations and determined future path oftravel, either estimated or measured and projected, it may be desirableto determine interactions between the determined path and the detectedobjects. Approaches that estimate a path of travel or measure the futurepath of travel may assume a path width based on either a typical lanewidth, such as 3.7 m, or based on the measured or estimated typicalwidth of the ego-car. Additionally, the future path may use laneestimation so that when lane markings are present the width and shift ofthe lane markings may be adaptively matched up to the computed orestimated future path of travel. In this example, when the vehicletravels in areas where the lane tracking algorithm temporary losestrack, such as in areas with lighting variations from overpasses, orareas facing toward the sun at sunrise/sunset, or fresh pavement wherelane markings have not yet been added, the path of travel alone maydetermine the lanes. Still, the determined location and width of thelanes may be more accurate due to the recent prior interaction with thelane detection algorithm. In some embodiments, a system may measure theoffset from the center of the lane and the width of a lane. In someembodiments, the lane detection system may include a tracking algorithmsuch as a Kalman Filter.

Atypical Traffic Events

Disclosed herein are methods and systems for determining the causes oftraffic events. In particular, this disclosure focuses on determiningthe causes of atypical traffic events and/or driver actions thatmitigate risk of such event. Although any act or event while driving avehicle may be characterized as an event, atypical traffic events asdescribed herein are notable because they may lead to some unsafecondition that has a higher probability of leading to an accident. Forexample, described herein are atypical events that do not occurfrequently (e.g. one out of a hundred minutes of driving may contain anatypical event for moderately rare events), but that may lead to unsafeconditions with relatively high frequency once they occur.

The systems and methods disclosed herein may determine whether a driveror control system of a vehicle is the cause of an atypical trafficevent, and may further determine whether such an atypical traffic eventis actually unsafe. For example, atypical traffic events as disclosedherein may refer to when one vehicle tailgates another or when a redlight is run by a vehicle. In certain scenarios, a driver or controlsystem of a vehicle may not be the cause of such atypical events, asdisclosed herein. Other atypical traffic events may be related to aninertial event, such as hard braking or accelerating. Other atypicaltraffic events may be manually input. For example, a traffic guard mayreport an atypical traffic event that a particular vehicle disregardedan instruction of the traffic guard when passing through anintersection. Accordingly, as used herein, an atypical traffic event canbe any event that may be unsafe that may have been caused by a driver orcontrol system of a monitored vehicle. When the systems and methodsdisclosed herein determine that a driver or control system of a vehiclewas not the cause of an atypical traffic event, the driver or controlsystem can be said to have performed permitted actions with respect tothe traffic event, may be said to have responded appropriately to thetraffic event, and the like.

As used herein, a monitored vehicle is a vehicle for which the systemsand methods disclosed herein determine causation for atypical trafficevents experienced by or at the vehicle. The vehicle may be monitored bysystems, sensors, processors, cameras, etc. installed on or in thevehicle. The vehicle may also be monitored by external cameras, sensors,etc. The data collected to monitor a vehicle can be processed by adevice in or on the vehicle, or by a remote device, as described herein.A monitored vehicle is any vehicle for which the systems and methodsdescribed herein determine causation with respect to atypical trafficevents.

In various embodiments, more than one vehicle at a time may be amonitored vehicle. For example, if a monitoring system is installed on afirst vehicle, the system may determine that the first vehicle is notthe cause of an atypical traffic event. However, it is contemplatedherein that the system may also determine that a second vehicle causedan atypical traffic event. In various embodiments, a system may reportonly that the first vehicle was not the cause of the atypical trafficevent, and/or may also report details relating to the second vehiclethat caused the atypical traffic event to a remote server or to anotherdevice nearby, such as a Wi-Fi enabled hard-drive affixed to a trafficpole.

For example, it may be desirable to collect data on other vehicles onthe road, particularly when those vehicles are the cause of atypicaltraffic events. Such information may be valuable in a number of ways.For example, the system could use the information to avoid certaindrivers or vehicle types in the future. In another example, the systemmay identify a vehicle type or fleet and report that information back toa manager of a fleet or other party associated with a second vehicle. Ina specific example, the vehicle monitoring systems disclosed herein maybe mounted on semi-trucks. The system may identify that a car serving asa car for hire (e.g., taxi, Uber, Lyft) was the cause of an atypicaltraffic event observed or experienced by a semi-truck with the drivermonitoring system installed on it. That information could be sold,licensed, or otherwise reported back to the party managing the car forhire, such that the information about the car for hire causing anatypical traffic event can be known by the managing party. In this way,data about vehicles or fleets that do not have the systems disclosedherein actually on board may still benefit from enabled devices that areinstalled elsewhere. Such vehicles like cars for hire may be identifiedin a number of ways using visual data, such as markings on the car,license plate numbers, stickers in the windows, etc. In anotherembodiment, a managing party that receives information about its driversmay provide information about its vehicles which can be used to identifythose vehicles on the road. In various embodiments, the drivermonitoring systems may not be installed on any vehicle, but may stillidentify and report on atypical traffic events and those who causedthem. For example, a driver monitoring system may be installed on atraffic light pole or fixture.

Detecting Driving Actions that Mitigate Risk—Tailgating

A traffic event may be an inertial event (such as a hard-braking event,a fast acceleration, a swerving maneuver, and the like), may be atraffic violation (such as failing to come to a complete stop at a stopsign, running a red light, crossing a double yellow line on a road, andthe like), may be defined by a person (such as a fleet safety managerdefining a traffic event through the specification of a time and/orplace of interest, a Driver indicating that unsafe driving is occurringin his or her vicinity, a traffic officer viewing a video feed remotely,and the like). In one example, a safety officer may specify a trafficevent as a period of time when a specified automobile passed through aspecific intersection, the specification of which may be based on areport of unsafe driving.

Traffic events may be caused by the ego-driver (who may be the driverbeing monitored), may be caused by another driver (who may be in thevicinity of the ego-driver), may be caused by something in theenvironment (such as road debris), or may have an unknown cause. Forexample, a traffic event may be specified as a time that the ego-driverslammed on the brakes. If the ego-driver slammed on the brakes becauseanother driver suddenly turned left in front of the ego-driver withoutsignaling, then the cause of the traffic event may be assigned to theother driver. If, however, the ego-driver slammed on the brakes so thathe could bring his car to a complete stop at a stop sign that had he hadfailed to notice earlier, but that had been clearly visible for sometime, then the cause of the traffic event may be assigned to theego-driver.

For systems and methods for detecting a driving action that mitigatesrisk in accordance with certain aspects of the present disclosure, theuse of many different types of sensors is contemplated. In the firstexample above, in which another driver turns left in front of theego-driver, a windshield mounted camera may be used to identify theother car. Alternatively, or in addition, RADAR and/or LiDAR may be usedto identify the other car. The movements of the other car may also bedetermined through data messages passed directly or indirectly betweenthe ego-driver's car and the other car that may indicate position and/orpath of travel for one or both cars. In addition, the movements of theego-car and the other car may be determined based on a stationary cameraor cameras that have a view on the scene of the traffic event, or may bebased on a camera that is affixed to a third car passing through thescene of the traffic event.

A car mounted camera may aid in this determination, but methods ofdetecting a driving action that mitigates risk that do not rely oncameras are also contemplated. For example, a method of determiningposition, such as GPS and/or dead-reckoning from a known location, inconjunction with previously or subsequently obtained information aboutthe position of a stop sign, may be used together to determine that thehard-braking event may have occurred in response to the stop sign at theknown or estimated stop sign position.

According to certain aspects of the present disclosure, detecting adriving action that mitigates risk may be rule-based, and/or may bebased on the output of a neural network trained on labeled data. Forexample, the output of a neural network may be used to identify othercars in the vicinity. FIGS. 3A-D and FIGS. 4A-D illustrate examples ofsystems and methods of detecting driving actions that mitigate riskbased on rules in combination with outputs of a neural network.Determinations of cause of traffic events based on rules and/or neuralnetworks may also be used to train a second neural network to detectand/or characterize traffic events and/or determine cause of a trafficevent.

FIG. 3A illustrates an example of detecting a driving action thatmitigates risk in which tailgating is detected and the cause is assignedto the ego-driver (or “Driver”). In the video frames shown in FIGS. 3A-Dand FIGS. 4A-D, the type of event and the determined cause is shown onthe top of the video frame, along with additional information. In FIG.3A, the type of event is “Tailgating”, the determined cause is “Cause:Driver” and the additional information is “From Front”.

In this example, certain aspects of the present disclosure were used todetect and track other cars, including a car that is in the same lane asthe Driver. In FIG. 3A, three cars travelling in the same direction asthe Driver are tracked. Each tracked car has an empty colored square,which may be called a “bounding box”, drawn over the location of thetracked car, along with a filled square containing information about thecar. In this example, a car ID is displayed in the top line of eachfilled square, and the distance to that car (in which the distance hasbeen converted to time based on the distance and the Driver's travellingspeed) is displayed in the next line of each filled square. From left toright, the tracked car IDs are “1”, “8”, and “0”, and the determinedfollowing distances in time is “0.7”, “1.6” and “0.5” seconds,respectively.

The distance may be determined, for example, based on methods describedabove and/or in the incorporated applications, or by other means, suchas by helicopter, autonomous aerial vehicle, smart pavement, and thelike. The determined speed of the driver's car, which may be used todetermine a following distance in time, is displayed at the top right ofthe video frame. In FIG. 3A, the determined speed of the Driver is 73miles per hour.

In one embodiment of the present disclosure, “tailgating” may beconfigured to mean that the Driver is travelling at an unsafe distancebehind another car in the same lane as the Driver. In another embodimentof the present disclosure, “tailgating” may be configured to mean thatthe Driver is travelling within a range of safe following distances atwhich the Driver's may benefit from reduced wind resistance and therebyimprove fuel efficiency. FIGS. 3A-D and FIGS. 4A-D illustrate examplesfrom an embodiment of the present disclosure in which “tailgating” isconfigured to mean that the Driver is traveling at an unsafe distancebehind another car in the same lane.

In FIG. 3A, the determined following distance to the car with ID “1”(car ID 1) is 0.7 seconds. A tailgating threshold may have beendetermined previously based on a user specification, and/or may be basedon data that indicates what may be considered a safe following distancefor a given driver and/or for a given road condition and/or for a givenregion. In this example, tailgating is configured to mean travelling atan unsafe distance behind another car and configured with a minimum safefollowing distance of 1 second. Since car ID 1 is in the same lane andis spaced at a following distance that is substantially below 1 second(0.7 seconds), a traffic event is detected. The bounding box around carID 1 may be colored red in a color display, and the filled square abovethe detected car may also be colored red, to identify which car is beingtailgated. In addition, a red/yellow/green indicator displayed in thetop center portion of the video frame shows a filled red circle, anunfilled yellow circle and an unfilled green circle. In the black andwhite drawing in FIG. 3A, the three indicator circles are depicted inblack and are distinguishable by position, with the red circle filledand on the left, the yellow circle in the middle, and the green circleon the right. In this configuration, the red/yellow/green indicatorindicates an unsafe driving situation (red, left), driving that is closeto the threshold of unsafe (yellow, middle), and safe driving (green,right), depending on whether the red, yellow, or green circle is filled,respectively. The red/yellow/green indicator in FIG. 3A indicates thatan unsafe traffic event (tailgating) has been detected.

As can be seen in FIG. 3A, other cars may be at an even shorterfollowing distance, but may not be considered participants in atailgating event. For example, car ID 0 is shown as having a followingdistance of 0.5 seconds. However, car ID 0 is not in the same lane asthe driver. Instead, car ID 0 is in the lane to the right of the driver.In this example, since the Driver's car is in the left most lane, andthere is a car within the threshold for tailgating in the right lane,the Driver does not have the option to change lanes at the time of thetailgating event.

In FIGS. 3A-D and FIGS. 4A-D, detected lane markings are drawn as heavyblack lines at locations corresponding to the left and right laneboundaries of the current lane of the Driver. In addition, thedetermined future path of travel, as determined according to certainaspects of the APP methods described above, is depicted as a grid thattapers to a narrower width from the bottom to the middle of the videoframe.

A short time after the event shown in FIG. 3A, the driver changes lanes.FIG. 3B shows the video frame captured from the camera attached to theDriver's car as the Driver is changing from the left lane to the laneformerly occupied by cars with IDs 0 and 8. As can be seen, car ID 0 hasalso moved over one lane since the video frame shown in FIG. 3A.

The detection of a lane change may be based on the locations of thedetected lane boundaries, as shown in heavy black lines. In FIG. 3B, theleft lane boundary intersects the bottom of the video frame near thecenter of the frame. The right lane marking would intersect the bottomof the video frame off to the right of the frame. Given the known orinferred orientation of the camera with respect to the body of the car(which may be inferred from the portions of the hood of the Driver's carin the bottom of the frame or which may be calibrated based on acquiredvideo frames) these lane boundary positions may be used to determinethat the car is crossing a lane boundary into another lane.

In addition, or alternatively, the relationship between the detectedlane boundaries and the vanishing point (VP) to which the detected laneboundaries point may be used to determine that the car is changinglanes. For example, the right lane boundary in FIG. 3A becomes the leftlane boundary in FIG. 3B. In FIG. 3A, the detected right lane boundaryintersects the bottom of the video frame at a horizontal position thatis substantially to the right of the vanishing point. In FIG. 3B,however, the same lane boundary (which is now the left lane boundaryinstead of the right lane boundary as it was in FIG. 3A) intersects thebottom of the video frame at a position that is substantially similar tothe horizontal position of the vanishing point. This change of therelative positions of lane boundaries and the vanishing point of thedetected lanes may be used to determine that a lane change has occurred.The detected lane change is indicated in the text box near the top ofthe video frame as “Lane Change: Detected—Direction: Right”.

In this example, after the lane change, the following distances of thethree tracked cars are 0.8, 1.2, and 0.5 seconds for the cars that arein the lane to the left of the Driver, in the same lane as the Driver,and in the lane to the right of the Driver, respectively. Because thecar that is in the same lane as the Driver (car ID 8) is more than 1second away from the Driver, there is no detected tailgating event inthis frame. Still, the following distance to car ID 8 is 1.2 seconds,which may be considered close to the threshold. The red/yellow/greenindicator in the center near the top of the frame therefore indicates a“yellow” state, meaning that the following distance is close to thethreshold for a traffic event. In this example, the Driver is exhibitingdriving behavior that is close to the configured threshold fortailgating.

A system or method in accordance with certain aspects of the presentdisclosure may determine that, while the traffic event (a tailgatingevent) was caused by the monitored driver, the monitored driverperformed an action (a lane change) that reduced the risk in thesurrounding environment. Still, an embodiment of the present disclosuremay be configured such that a risk mitigating action of this type is notrecognized as a “Star Alert.” For example, the system may be configuredso that “Start Alerts” are suppressed when the corresponding trafficevent to which the driver's action was responsive was also caused by themonitored driver.

As shown in the video frame in FIG. 3C, which was captured a short timeafter the video frame shown in FIG. 3B, the Driver has been drivingfaster than car ID 8, and the following distance has decreased to 1.0seconds. A short time later, as shown in the video frame in FIG. 3D, thefollowing distance has now decreased to 0.9 seconds, and once again atailgating event is identified. The cause of the traffic event(tailgating) shown in FIG. 3D is assigned to the Driver according to aconfigurable set of rules that are described in the next section.

In the example driving scenario illustrated in FIG. 3, the driverperformed one action (a lane change) that had an effect of mitigatingrisk in the environment. This action, however, was sandwiched betweentwo risky traffic events that could also be attributed to the driver. Asystem or method for determining actions that mitigate risk maydetermine that the driver is alert and responding to the environment butmay further determine that the behavior of the driver is aggressive. Todetermine whether the driver's behavior should be recognized withpositive reinforcement, such as with a “Star Alert”, a comparison ofdriving risk after a driver's action may be compared with the predictedrisk of a typical driver. A typical driver, for example, may not be anaggressive driver. In this example, at the time corresponding to thescene illustrated in FIG. 3C, a system may determine that a typicaldriver would not only have changed lanes, but would have also createdmore space between himself and other cars on the road. In someembodiments, a driving action that is at least as safe as a typicaldriver in the same scenario may be a criterion for the awarding of aDriverStar.

FIGS. 4A-D illustrate an example of a tailgating event that is not thecause of the Driver. FIG. 4A, which depicts a video frame captured about30 seconds after the video frames that are depicted in FIGS. 3A-D, showsthat the driver is in the far-right lane with no visible cars in thesame lane. Car ID 0 and car ID 36 are detected and tracked in the laneto the left of the Driver. The distance to car ID 0 is 0.4 seconds andthe distance to car ID 36 is 1.2 seconds. We can infer from thesefollowing distances that car ID 0 is tailgating car ID 36, since thedifference is following times is (1.2 seconds-0.4 seconds=0.8 seconds)is substantially less than the configured threshold for tailgating (1.0seconds). Still, the Driver in this example, which is the driver of thecar to which the camera collecting the video frame is attached, is notexhibiting tailgating behavior. The red/yellow/green indicator,therefore, is in a green state (right-most black circle is filled),indicating that the Driver is exhibiting good driving behavior at thistime.

According to certain aspects of the present disclosure, the Driver maybe alerted that a car in an adjacent lane (for example, car ID 0 in FIG.4A) is exhibiting unsafe driving behavior (tailgating), and/or mayreceive a positive reinforcement or positive driving assessment foravoiding the unsafe driver. The Driver may avoid and unsafe driver, forexample, by slowing down and thereby increasing the following distanceto that car, even though the that car is in an adjacent lane.Furthermore, an embodiment of the present disclosure may report theunsafe driving of another driver by sending a message to a centralizeddatabase. In addition, or alternatively, in the case that the observedcar is a part of a recognized fleet of automobiles (such as a truckingline or a ride sharing network such as Uber or Lyft), an embodiment ofthe present disclosure may report the unsafe driving event to a safetymanager responsible for the fleet of automobiles or to other interestedparties.

FIG. 4B, which includes a video frame that was captured shortly afterthe video frame shown in FIG. 4A, shows that car ID 0 has signaled anintention to change lanes into the monitored driver's lane.

FIG. 4C, which includes a video frame that was captured shortly afterthe video frame shown in FIG. 4B, shows that car ID 0 has entered theDriver's lane from the left. The text banner near the top of the imageindicates: “Tailgating: Detected—Cause: Other Car—From Left”. Thered/yellow/green indicator is in the red state, indicating the unsafedriving event. Unlike the example in FIGS. 3A and 3D, however, thisunsafe driving situation was caused by the other car and not the Driver.In some embodiments of the present disclosure, events for which thecause is not attributed to the Driver may not impact an overall ratingof the Driver's driving behavior. Additionally, video data associatedwith such events may not be transmitted wirelessly to a safety manageror other interested party in consideration of bandwidth or storagecosts. An alternative embodiment of the present disclosure, however, maybe configured so that a certain number of events caused by other driversmay negatively impact a Driver's overall driving behavior assessment, asa number of such events may indicate that the Driver tends to puthimself or herself in the vicinity of unsafe driving events.

FIG. 4D, which includes a video frame that was captured several secondsafter the video frame shown in FIG. 4C, shows that the Driver is stilltailgating car ID 1, now at a following distance of 0.5 seconds. Whilethe following distance has increased from 0.4 seconds to 0.5 seconds,the speed of the Driver has also increased from 76 mph to 80 mph. Inaccordance with a configuration of tailgating rules that are describedin the next section, a new tailgating event is now detected and thecause is attributed to the Driver. In this configuration, if atailgating event that is caused by another driver continues for morethan 5 seconds, a new tailgating event is triggered that is nowattributed to the Driver. In this example, when the Driver gets cut off,the cause may be initially attributed to the other driver. The Driverthen has some time (for example, 5 seconds for the time that the otherdriver first entered the Driver's lane) to respond and create anappropriate following distance. If the Driver does not, then the causefor the persistent tailgating may be assigned to the Driver.

In some embodiments of the present disclosure, the relative speeds ofthe Driver's vehicle and another vehicle may be compared. In addition,the pedal actions (which may be determined from a bus (such asOBD2/CANBUS/J1939)), or may be inferred based on visual information, maybe used to modify rules for determining cause. In one embodiment, thethresholds for determining the presence and/or severity of tailgatingmay be modified based on the relative speed of the cars. For example, ifa Driver gets cut off but the relative speeds of the cars are such thatthe separation is increasing, then the Driver may be given more time toslow down or change lanes before a persistent tailgating event isassigned to the Driver. Similarly, if the pedal action or fuel flow ofthe Driver's car indicate that the driver has taken positive action toincrease the following distance, then additional time may be allowed.

While FIGS. 3A-D and FIGS. 4A-D illustrate a method of determining anaction that mitigates risk that is responsive to tailgating events, theactions that are responsive to other traffic events are alsocontemplated. Furthermore, while FIGS. 3A-D and FIGS. 4A-D illustratethe functioning of an embodiment of the present disclosure that relieson a camera sensor, other sensors are contemplated. Embodiments that donot include camera sensors are also contemplated. For example, thedetection of a traffic event may be based on inertial sensors, whichmay, for example, indicate a hard stop. In this case, the cause of thehard stop may be determined with the aid of a camera if, for example,the hard stop was caused by the sudden presence of a pedestrian or abicyclist. A camera may not be necessary, however, if the hard stop iscaused by a red light, and the driver's car records the presence of thered light based on a data message broadcast from the traffic light.

While FIGS. 3A-D and FIGS. 4A-D illustrate an example in which a lanechange was determined based on visual cues, some embodiments of thepresent disclosure may infer lane changes based on inertial sensorsand/or based on readings of a signaled lane change from the Driver's carand a corresponding movement of the steering wheel.

Detecting Driving Actions that Mitigate Risk—Hard-Braking, Pedestrian

FIGS. 5A-D illustrate an example of a hard-braking event that is causedby a road condition. In this example, the hard-braking event is causedby a pedestrian crossing the street at an unexpected location. FIGS.5A-D illustrate annotated images captured by a camera located inside ofa truck and facing forward.

FIG. 5A illustrates an apparently normal driving scene with two lanes oftraffic travelling in the direction of the monitored Driver, a centralturn lane, and two lanes in the on-coming direction. On this wide road,the traffic is moving at a fast speed, with the monitored Drivertravelling at 62 MPH. In FIG. 5B, a pedestrian can be discerned crossingthe street. At this point the pedestrian is directly in front of theDriver. According to a distance estimate of a nearby vehicle, the Driverwill arrive at the location of the pedestrian in 2.0 seconds. The Driverhas already reduced his speed to 52 MPH and is still braking.

At the time that the image of FIG. 5C is captured, the monitoredDriver's speed has further reduced to 45 MPH. The Pedestrian has almostcleared the monitored Driver's lane. The peak hard-braking forced wasmeasured as 0.45G. Shortly thereafter, as illustrated in FIG. 5D, thePedestrian has cleared the Driver's lane and has entered the middle turnlane.

In the example shown in FIGS. 5A-D, a traffic event may be attributableto an unexpected road condition, such as the sudden appearance of aPedestrian at a location far removed from a crosswalk. In this example,the cause of the hard-braking event may not be assigned to the Driver,since the Driver responded appropriately to the unexpected and atypicaltraffic event. According to certain aspects of the present disclosure,the determined cause of the event may impact a summary score of theDriver differently depending on the determined cause. For example, theimpact on the Driver's score may be more positive than it would be ifthe determined Cause were the monitored Driver. In some embodiments, theimpact on the monitored Driver's score as a result of the determinedevent may be positive overall. In the case of avoiding a collision witha Pedestrian, a successfully executed hard-braking maneuver may indicatealert and proactive driving. In this view, the hard-braking maneuver maybe an event that a fleet manager or the like would want to acknowledgeand encourage.

Some driver monitoring systems may focus on identification ofproblematic and/or risky driving. For example, a driver monitoringsystem based on inertial sensor reading may identify driving maneuversthat are subsequently labeled reviewed by a human operator and thenreported if the operator determines that all of the criteria of theproblematic and/or risky driving behavior were met. In one example, aninertial sensor reading may indicate a hard-braking event. Videoassociated with the hard-braking event may then be transmitted to ahuman operator. The human operator may then reject alerts correspondingto sequences like the one just described. That is, for a systemconfigured to find examples of negative driving, an example of positivedriving may be mistakenly detected based on an inertial signature. Suchan alert is then typically suppressed by a human operator who may reviewthe corresponding video footage to reject “false alarms”. Accordingly,with currently available driver monitoring systems, any‘above-and-beyond’ driving maneuvers may only be reported throughvisible bystander eyewitness account. Such accounts may occur at a verylow frequency.

Systems and methods in accordance with the present disclosure however,may positively recognize a traffic sequence such as the one illustratedin FIGS. 5A-D as an example of good driving.

According to certain aspects of the present disclosure, a drivermonitoring system may enable improved visibility into the day of adriver. For example, an embodiment of the present disclosure may captureand analyze driving event when risky driving or risky trafficenvironments are not identified. The system may positively determinethat the driver is driving in a low risk environment, as may accordinglyassign a rating of “safe” to these periods of driving.

There may be many instances in which a driver, who may be a professionaldriver, may have deep domain experience and may actually cause thedriving environment around them to be less risky for themselves and forother drivers. That is, a driver may perform driving maneuvers that arenot only safe (have little or no risk), but, furthermore, through theexecution of the action the driver may have created a safer drivingenvironment for all vehicles within the immediate cluster.

In the example illustrated in FIGS. 5A-D, the monitored driver's alertbraking may have caused other drivers to take notice of the pedestrianwho was jaywalking across the road. According, a “Star Alert” may begenerated responsive to the detected driving action or sequence ofactions that serves to improve the immediate driving environment.

Detecting Driving Actions that Mitigate Risk—Hard-Acceleration,Emergency Vehicle

FIGS. 6A-D illustrate an example of an atypical hard-acceleration eventthat is caused by an emergency vehicle. FIGS. 6A-D illustrate annotatedimages captured by a camera located inside of a car and facing forward.FIG. 6A illustrates a normal driving scene. The monitored Driver is inthe left lane of a two-lane road that is separated from on-comingtraffic by a cement divider. The road leads to a highway and has a wideshoulder. The Driver is travelling 52 MPH.

The image in FIG. 6B indicates that the Driver has changed lanes twiceto get over to the shoulder. In addition, he has been braking steadilyfor several seconds and is now travelling 34 MPH. The Driver did notbrake hard enough to trigger the detection of a hard-braking trafficevent and therefore may have been missed by an inertial-trigger baseddriver monitoring system. In FIG. 6C the Driver is starting toaccelerate again from rest, traveling 5 MPH on the shoulder. The causeof the Driver's maneuver is now visible as a fire truck races by. InFIG. 6D, now that the fire truck has safely passed, the monitored Driverhas returned to the road. At this point, the monitored Driver has toquickly regain Highway speed. In doing so, he accelerates with adetected maximum force of 0.44 G. According to previously determinedthresholds, this triggers a hard-acceleration traffic event.

There are several methods contemplated to determine whether thishard-acceleration event is responsive to determined traffic event. Formethods that rely on visual data, a system in accordance with thepresent disclosure may detect the fire-truck and determine that it is anemergency vehicle. In one example, the detection of a fire-truck at atime shortly after the hard-braking event may indicate that the cause ofany detected traffic event could be attributable to the fire-truck. Inanother example, the presence of the emergency police vehicle may bedetermined based on an audio signal recorded by a microphone on a drivermonitoring device. In another example, the presence of the emergencyvehicle may be communicated via a dedicated short-range communications(DSRC) protocol. In another example, the pattern of driving behaviorsexhibited by other visible or otherwise detectable drivers may be usedas the basis for determining that the monitored Driver was responding tothe presence of an emergency vehicle. In this example, the trajectoriesof other detected and tracked cars may be consistent with traffic makingroom for an emergency vehicle to pass. Because the hard-braking eventappeared a short time before the hard-acceleration event, the driver maybe excused for the hard-acceleration.

Furthermore, in accordance with certain aspects of the presentdisclosure, the monitored driver may be recognized as having performed apositive “Star Alert” when he pulled over in response to the fire-truck.In this example, the traffic event to which the driver's action wasresponsive may have occurred prior to the time that the unsafe trafficevent (e.g. a hard braking event) could have been detected. For example,the presence of the emergency may not have been detected by the systemuntil the emergency vehicle passed by the vehicle. In this example, thesystem may first detect the action of the driver and then determine thatit was responsive to an traffic event that was detected at a later time.

Learning to Detect Actions that Mitigate Risk

According to certain aspects of the present disclosure, detecting adriving action that mitigates risk may be based on a learning-basedcausation model. According to certain aspects, a multi-layer perceptronmay be trained on supervised training data to generate risk levellabels. Alternatively, a video caption generation system may be trainedon a series of frames. The video capture generation system may be basedon a Recurrent Neural Network (RNN) structure, which may use LongShort-Term Memory (LSTM) modules to capture temporal aspects of atraffic event.

The data used to train a learned model may be generated by a rule-basedapproach, such as described above. These labels may be accepted,rejected, or corrected by a human labeler. According to certain aspects,inputs from fleet safety officers may be utilized. For example, a fleetsafety officer may correct a given action responsivity label, or mayagree with labels that are provided by a rule-based and/or neuralnetwork based system. These labels may then be used to bootstrap fromthe rule based approach to a machine learned model that exhibitsimproved performance.

Driver Safety Monitoring in Fleets

In the United States, the Occupational Health and Safety Administration(OSHA) may require that employers provide their employees with a safeworking environment and comply with federal safety regulations. WhileOSHA may enforce these regulations through worksite inspections andrecording requirements, it often falls to employers to create anorganizational climate that prioritizes safety, even in the face ofcompeting organizational goals. A positive safety climate may exist whenemployees believe that safety is valued and expected by theirorganization. A positive safety climate may lead to increased safetyperformance and, as a result, reduced workplace accident and injuryrates. Conversely, a poor safety climate may be linked to increases inworkplace accident and injury rates as well as accident underreporting.Research emphasizes that managerial commitment is key to the promotion,maintenance, and reinforcement of a positive safety climate. All levelsof management, from senior executives to frontline supervisors, maypublicly commit to, communicate, and treat safety as a priority. Theymay effectively do so through a safety program that is designed torecognize and reward appropriate safety behaviors.

Fleets may recognize and reward drivers based on time/mileage basedmilestones. For example, a Safety Award Program may recognize driverassociates who operate without a preventable accident. Awards, which mayinclude pins, hats, patches, and jackets, may be given after 6 months, 1year, and then every year thereafter.

Existing driver monitoring solutions may focus on reducing the frequencyof negative driver behavior events. With these systems, a fleet managermay identify risky driving behavior and may seek to correct drivingskills through coaching. Current driver monitoring system however, maynot consider positive factors relating to driving compliance, positiveperformance, good driving, and a driver's execution of an action or setof actions responsive to a detected traffic event that has the effect ofmitigating risk.

Encouraging Good Driving Behavior

According to certain aspects of the present disclosure, a drivermonitoring system may consider positive factors. These factors maycontribute to a system of encouraging good driving behavior. Anembodiment of certain aspects of the present disclosure may be referredto as DRIVERI™. A DRIVERI™ system may serve as a driver advocate, byproviding fleets with systems and methods to recognize and reward theirdrivers for exhibiting good driving behavior.

Current driver monitoring systems may equate an absence of negativedriving event detections with good driving. However, time spent outsideof dangerous events may be made up of many moments of at-risk drivingthat are not dangerous to varying degrees. For example, there may be arange of driving between safe driving and proactively courteous driving.

A system that analyzes driving behavior based on the absence of negativetraffic events may not accurately consider time that does not includedriving at all. According to some systems, a driver who experiences onenegative driving event in twenty minutes of driving may be considered assafe as a driver who experiences one negative driving event over thecourse of an eight-hour driving shift. Furthermore, systems that arebased on the detection of negative driving event may emphasize the mostdangerous driving behaviors exhibited by a driver, and may fail toproperly recognize and reward safe and courteous driving.

Certain aspects of the present disclosure may be applied to createawareness of the portion of a driver's day that may be under-recognizedby existing driver monitoring technologies. A system in accordance withthe present disclosure may affirmatively analyze time intervals toquantify different aspects of safe driving. A time interval of safedriving may be defined not just as an absence of a negative event (suchas a hard-braking event), but instead may be defined based on a numberof pre-configured criteria. Examples of such criteria includemaintaining proper lane position, using lane change indicators, makingroom for vehicles stopped on the shoulder, and the like. If a driver isexhibiting all of the desired features, he may be considered to bedriving in the GreenZone™.

Unlike a system based on detecting negative events alone, a GreenZone™system may serve as a basis for positive reinforcement. For any systemof driver monitoring, it may be desirable to increase good, safe, andcourteous driving, and also decrease negative driving events, such ascollisions, hard-braking, and the like. A system based on punishment fordetected negative behaviors may attempt to stop negative drivingbehaviors. Such a system, however, may not encourage specific gooddriving behaviors. In addition, such a system may fail to recognizeat-risk driving situations which do not result in a negative drivingevent. While being in a situation in which an accident is likely may notbe as undesirable as being in an actual accident, for a driver whofrequently finds himself in situations for which an accident is likely,it may only be matter of time.

By focusing on positive behaviors, instead of or in addition to negativebehaviors, the dynamic between a fleet manager and a driver may change.Rather than focusing exclusively on collisions and near-collisions, withGreenZone™ monitoring, a fleet manager may be able to point out expertmaneuvers by expert drivers in the fleet. Such recognition maystrengthen the relationship between excellent drivers and a truckingcompany. In addition, examples of excellent driving may be used toinstruct less experienced drivers.

In some of the traffic scenarios described herein, if the monitoreddriver fails to adjust to driving conditions, the DRIVERI™ system mightreport the risky driving condition as a negative event. According tocertain aspects of the present disclosure, however, additional contextof the driving environment and/or determination that the driverperformed an action that mitigated the risk may cause the same event tobe classified as a ‘StarAlert’.

In addition, a DRIVERI™ system, or other system in accordance with thecertain aspects of the present disclosure may track at-risk but notdangerous driving. These instances may be valuable as coachable moments.

A calculation of a GreenZone™ score may be based on the number ofminutes driven in a day in which a dangerous event was detected, inwhich an at-risk event was detected, and during which the drivingbehavior met the full set of positive driving criteria. In someembodiments, a GreenZone™ score may be further based on exhibitedexemplary behaviors, which may be “above-and-beyond” the expectations ofa typical safe driver.

In one embodiment, a DRIVERI™ system may continuously record video andother sensor data while a vehicle is running. In one example, the videoand other data may be segmented into 1 minute durations. Based on a 1min video duration and 100% duty cycle, an eight-hour driving day maygenerate 480 1-minute videos. A driver may not be active for eight hourscontinuously. In these cases, the number of recorded videos may be less.The recorded videos may be analyzed with a DRIVERI™ service. A recordedvideo may be analyzed using processors embedded within a device in thevehicle and/or by one or more processors in the cloud. In someembodiments, the processing capabilities of embedded processors may notbe able to analyze all the recorded video as fast as it is collected. Inthis case, some of the recorded minutes may be ignored. In anotherembodiment, a processor embedded with the vehicle may process the visualdata in a streaming fashion.

Several systems and methods of determining causation of traffic eventsand encouraging good driving behavior are described in PCT applicationPCT/US17/44755, entitled “DETERMINING CAUSATION OF TRAFFIC EVENTS ANDENCOURAGING GOOD DRIVING BEHAVIOR”, filed 31 Jul. 2017, which isincorporated herein by reference in its entirety.

Traffic Incident Reports

For some traffic events, such as traffic events belonging to aparticular class, or traffic events corresponding to a particularvehicle and/or time and/or location of interest, it may be desirable tosee a traffic report. In accordance with certain aspects of the presentdisclosure, a traffic incident report may be automatically generated.Traffic incident reports may be generated for positive, proactivedriving behaviors as a way of acknowledging and reinforcing positivedriving behaviors. Other traffic incident reports may be generated forminor collisions and the like, which may assist a fleet manager'sresponse to the event.

FIG. 7 illustrates a traffic event report that was generated in responseto a reported accident. In the incident that is the subject of thereport, a truck came into contact with a car that was idling ahead of itat a traffic-light. A traffic incident report of such an incident may begenerated based on a user's request, corresponding to a user-providedalert-id, which may be referred to as an incident-id. In anotherexample, a report may be generated based on an automatically detectedtraffic incident without additional input from a person.

The report illustrated in FIG. 7 includes information relating to themonitored driver at a number of time-scales. A cartoon having abirds-eye view may illustrate the positions of one or more vehicles in ascene for a duration corresponding to a 1-minute period around theaccident that is the subject of the report. In some embodiments, astatic representation of the report may include a last frame of thecartoon. A corresponding animated video may be available as well. Forexample, clicking on the cartoon image may enable the user to view acartoon video of the event.

The report illustrated in FIG. 7 also includes a timeline ofgreen-minutes and alerts by this driver on a day. This information panelmay correspond to a day-long time-scale. The report illustrated in FIG.7 also includes a trend of the daily driver score for this week andsummary of alerts. This information panel may correspond to a week-longtime-scale. The report illustrated in FIG. 7 also includes a summary ofthis driver's behavior benchmarked against himself from the previousweek, and against the fleet average. The benchmarks may includecomparisons based on fine-grained assessments of driving conditions,such as a comparison versus fleet-wide statistics at certain times ofday (at night, in the morning heavy-traffic, and the lie). In someembodiments, the text of an accident report may be made editable.

For the particular incident that is the subject of the reportillustrated in FIG. 7, the various time-scale information panels mayoffer additional insight into the context of the accident. Theinformation panels may incorporate information gleaned from outwardfacing cameras, inward facing cameras, inertial sensors, sensor dataavailable via the OBD port and the like. In this example, the reportreveals that the driver ran a red traffic light 45 mins prior to theincident. In addition, the driver consistently had risky/moderate eventsthat led up to the reported incident. Based on the inward-facing camera,it can also be appreciated that the Driver was looking away from theroad for a substantial portion of the minute that contained theaccident. The driver-score trend is also consistent with theseobservations.

Based on the provided accident report, one may make inferences that mayplace a Driver in a better position for success. For example, one mayconclude that this driver is not a good “morning-person”, but that he isan above average driver at night. In addition, or alternatively, theinformation selected for the report may be used to determine that thedriver was having an unusually bad day. In this case, the trafficincident may have been avoided if there was an intervention (maybe 30minutes prior), around the time that the Driver's rolling summarydriving score can be observed to have decreased.

An accident report generated in accordance with certain aspects of thepresent disclosure may be useful for enabling timely notifications thatmay prevent avoidable accidents. In addition, the strength of risk, andor number of moderate incident counts may be used for predictingaccidents which may enable for personalized insurance premiums.

As used herein, the term “determining” encompasses a wide variety ofactions. For example, “determining” may include calculating, computing,processing, deriving, investigating, looking up (e.g., looking up in atable, a database or another data structure), ascertaining and the like.Additionally, “determining” may include receiving (e.g., receivinginformation), accessing (e.g., accessing data in a memory) and the like.Furthermore, “determining” may include resolving, selecting, choosing,establishing and the like.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: a, b, or c” is intended to cover: a, b, c,a-b, a-c, b-c, and a-b-c.

The methods disclosed herein comprise one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isspecified, the order and/or use of specific steps and/or actions may bemodified without departing from the scope of the claims.

The processing system may be configured as a general-purpose processingsystem with one or more microprocessors providing the processorfunctionality and external memory providing at least a portion of themachine-readable media, all linked together with other supportingcircuitry through an external bus architecture. Alternatively, theprocessing system may comprise one or more specialized processors forimplementing the neural networks, for example, as well as for otherprocessing systems described herein.

Thus, certain aspects may comprise a computer program product forperforming the operations presented herein. For example, such a computerprogram product may comprise a computer-readable medium havinginstructions stored (and/or encoded) thereon, the instructions beingexecutable by one or more processors to perform the operations describedherein. For certain aspects, the computer program product may includepackaging material.

Further, it should be appreciated that modules and/or other appropriatemeans for performing the methods and techniques described herein can bedownloaded and/or otherwise obtained by a user terminal and/or basestation as applicable. For example, such a device can be coupled to aserver to facilitate the transfer of means for performing the methodsdescribed herein. Alternatively, various methods described herein can beprovided via storage means (e.g., RAM, ROM, a physical storage mediumsuch as a thumb drive, etc.), such that a user terminal and/or basestation can obtain the various methods upon coupling or providing thestorage means to the device. Moreover, any other suitable technique forproviding the methods and techniques described herein to a device can beutilized.

It is to be understood that the claims are not limited to the preciseconfiguration and components illustrated above. Various modifications,changes and variations may be made in the arrangement, operation anddetails of the methods and apparatus described above without departingfrom the scope of the claims.

What is claimed is:
 1. A method comprising: detecting, by at least oneprocessor of a computing device in communication with a camera, thepresence of a first vehicle in a field of view of the camera, whereinthe camera is mounted on or in a second vehicle; predicting, by the atleast one processor, that the first vehicle will move into a lane inwhich the second vehicle is traveling; determining, by the at least oneprocessor, that the first vehicle moved at least partially into the laneat a first point in time, wherein determining that the first vehiclemoved at least partially into the lane comprises: detecting, by the atleast one processor, the presence of the first vehicle in the field ofview of the camera at the first point in time; determining, by the atleast one processor, a bounding box that surrounds the first vehicle invisual data captured by the camera at the first point in time; anddetermining, by the at least one processor, a location of a lane line inthe visual data where the lane line intersects a bottom of the boundingbox; determining, by the at least one processor, that the second vehicleslowed down or changed lanes within a predetermined amount of time fromthe first point in time; and determining, by the at least one processor,that a driver of the second vehicle mitigated a predicted risk based onthe prediction that the first vehicle will move into the lane, thedetermination that the first vehicle moved into the lane, and thedetermination that the second vehicle slowed down or changed lanes. 2.The method of claim 1, further comprising determining a predictedfollowing distance between the second vehicle and the first vehicle,wherein the predicted following distance is below a predetermined safetythreshold, and where the predicted risk is based at least in part on thepredicted following distance.
 3. The method of claim 2, wherein thesecond vehicle slowed down within the predetermined amount of time, andwherein determining that the driver of the second vehicle mitigated thepredicted risk further comprises determining an actual followingdistance between the second vehicle and the first vehicle, wherein theactual following distance is above a predetermined safety threshold. 4.The method of claim 1, wherein the second vehicle slowed down after thefirst point in time, and wherein determining that the driver of thesecond vehicle mitigated risk further comprises determining a followingdistance between the second vehicle and the first vehicle at a secondpoint in time, wherein the second point in time is subsequent to thefirst point in time.
 5. The method of claim 1, wherein the secondvehicle slowed down or changed lanes before the first vehicle moved atleast partially into the lane.
 6. The method of claim 1, whereindetermining that the driver of the second vehicle mitigated a predictedrisk further comprises determining a following distance between thesecond vehicle and the first vehicle at a point in time before the firstpoint in time at which the first vehicle moved at least partially intothe lane.
 7. The method of claim 1, further comprising: positivelyadjusting a driver grade for the driver based on the determination thatthe driver of the second vehicle mitigated the predicted risk.
 8. Themethod of claim 2, further comprising determining, by the at least oneprocessor, the predetermined safety threshold at least in part based ona speed of the second vehicle.
 9. The method of claim 8, wherein thepredetermined safety threshold is lower when the speed of the secondvehicle is higher.
 10. The method of claim 1, further comprisingtransmitting, by the at least one processor, an alert notification to asecond device, wherein the alert notification includes an indicationthat the driver took action to mitigate driving risk.
 11. The method ofclaim 10, further comprising transmitting, by the at least oneprocessor, visual data to a remote cloud server, wherein the visual datacomprises data captured by the camera when the second vehicle sloweddown or changed lanes.
 12. The method of claim 11, further comprisingtraining an autonomous vehicle driving system with the visual data. 13.The method of claim 1, wherein the first vehicle moved at leastpartially into the lane from a merging lane of a highway.
 14. The methodof claim 13, further comprising: detecting, by the at least oneprocessor, a dotted lane line, and wherein predicting that the firstvehicle will move into the lane is based on at least the detected dottedlane line.